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ABSTRACT 


The final technical report for contract NAS3-25279, Reusable Rocket Engine 
Turbopump Health Monitoring System, is presented. The work Breakdown 
Structure is correlated to the nature of the actual tasks performed. Task 1, 
degradation mechanisms, and lasted., sensor identification/selection resulted 
in a list of degradation modes and a list of sensors that are utilized in the 
diagnosis of these degradation modes. The sensor list is divided into primary 
and secondary indicators of the corresponding degradation modes. The signal 
conditioning requirements are discussed, describing the methods of producing 
the SSME post-hot-fire test data to be utilized by the Health Monitoring 
System. 


Development of the diagnostic logic and algorithms is also presented. The 
knowledge engineering approach, as utilized, includes the knowledge 

acquisition effort, characteri zat ion of the expert's problem solving strategy, 
conceptually defining the form of the applicable knowledge base, and rule 
base, and identifying an appropriate inferencing mechanism for the problem 
domain. The resulting logic flow graphs detail the diagnosis/prognosis 
procedure as followed by the experts. The nature and content of required 
support data and databases is also presented. The distinction between deep 
and shallow types of knowledge is identified. Computer coding of the Health 

Monitoring System is shown v to)fo How the logical inferencing of the logic flow 
graphs/algorithms . Coding- ....-is- performed using both the conventional 

programming language "C", and an expert system development tool. The computer 
code was delivered to LeRC as part of this program. Finally, an HMS 
development plan is presented, detailing suggested HMS enhancements to 
increase its functionality and robustness. 
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INTRODUCTION 


The final technical report for contract NAS3-25279, Reusable Rocket Engine 
Turbopump Health Monitoring System, will follow the same descriptive format as 
the monthly reports. Each task of the program will be discussed in a 

sequential manner as it was satisfied during the program effort. Tasks 1 and 

2 provided the list of sensors and the failure modes that were used in the 

diagnostic system along with a methodology for determining failed sensor 
characteristics . Initial efforts in lask 3 concentrated on developing a set 
of logic flow graphs that captured the domain experts heuristic knowledge of 
how turbopump diagnostics are performed. The final activity of lask 3, 

algorithm development, was completed iri conjunction with lask 4 , data 
processing. Data sampling arid processing procedures that are performed on all 
SSME test and flight data were reviewed and evaluated for use in this 

program. This information was then used in the final development of the 
system algorithms . 


Once the algorithms had been developed, they were then coded as a functional 
whole comprising the diagnostic system for this program. The coding 
constituted a portion of the activity in lask 5. Iwo methods of coding the 
algorithms were used. One set is coded in the "C" language with another set 
in an expert system shell called 02. Both systems capture the same knowledge 
and provide the same diagnostic capability. The technical activity for lask 
5, and for the entire program, cumulated in an HMS conceptual design and 
development plan. lhe design and development plan gives an overview of the 
system that has been developed and presents a detailed procedure for expanding 
the existing diagnostic system to include multiple failure modes, prognostics, 
and life prediction. 


Health Monitoring System Development 


Initial activity for lask 1, degradation mechanism, and lask 2, sensor 
identification/selection, started with a review by turbomachinery personnel of 
the life-limiting components that had previously been generated in a 1983 
study. Reusable Rocket Engine lurbopump Condition Monitoring, NAS3-233T9 . 
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Specific HP01P life-limiting component degradations that were discussed in 
this study were modified to reflect current pump design and engine hot fire 
test experience that had been gathered since 1983. In conjunction with the 

degradation mode identification, updates to the sensor technology as given in 
the 1983 study were made to incorporate experience gained. After the updates 
had been completed, a preliminary listing of life-limiting component 

degradation modes and sensor technologies was established. Discussions of 
defective sensor detection, Task 1: Sensor Methodology, are given along with 

the discussion of algorithm development. 

Further discussions with the domain experts finalized the list of failure 
modes and sensors. Appendix A includes a list of the failure modes, lable I, 
and Primary Identification (P1D) Numbers, Table 2, that were used in this 
program. The PIO numbers are the SSMF flight and facility instrumentation 
identifiers. Redline measurements are identified, and where appropriate, are 
broken into primary and secondary listings. Those measurements listed as 
primary are the first line indicators of a particular failure mode. Secondary 
measurements are those that the turbopump experts review in conjunction with 
the primary measures to aid in the interpretation of turbopump performance. 

It is not to be inferred that information derived from the measurements listed 
with a particular failure mode give an unequi vocable indication that the 
associated turbopump component has failed. The redline measurements are 
certainly used in this fashion; however, several of the other components do 
not have direct measures associated with them and inferences must be made. An 
example is ball bearing wear. Here, accelerometers are sources of dynamic 
data which contain certain operating frequencies that are indicators of 
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bearing performance. These frequencies provide a screening measure from which 
the turbopump experts deduce what has happened during the test with regard to 
the bearing. The topic of additional and advanced sensor requirements is 
addressed in the HMS Development Plan. 


lhe second subtask within Task 2 was signal conditioning requirements. 
Appendix B gives a detailed description of the signal conditioning 

requirements for this program. Since the sensors used for this diagnostic 
system are the same as those used on existing SSMT turbopumps, the data 
processing follows the same procedures and policies that are followed by 
Rocketdyne in the data reduction of all SSMK test and flight data. For this 
reason, Appendix B not only prescribes the signal conditioning requi rements 
but also specifies the data processing requirements and identifies the dala 
sampling rate, lask A. To briefly summarize Appendix B, the digital tapes are 
read by a Perkin-Elmer computer system, converted into 32 bit packed binary 
floating point representations, and stored on a 9-track tape. Rocketdyne 
generated computer programs are then used to selectively extract data and 
perform a variety of manipulations. These results are then made available as 
time-history plots of sensor values as well as stored on magnetic tape as 
sensor data vs. time. 


The analog recording and processing system, which includes data from 
accelerometers and strain gauges, stores the data in continuously varying 
analog voltage form on PM tapes. A Fl-I data reduction technique is applied to 
the data resulting in Fourier Coefficients characterizing the signals in the 
frequency domain. Further processing typically includes the generation of 

power spectral density plots, giving the square of acceleration divided by 
frequency vibration sensed at varying frequencies, including the resonant 
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peaks corresponding to the vibration modes present within the signal. This 
data can be made available as x-y point values, the format for plotting stored 
on magnetic tape. The HMS diagnostic system utilizes both of the tapes 
generated by the digital and the analog recording methodology. 

Having identified the failure modes and sensors that would be used for this 
system the process of logic and algorithm development began. A knowledge 
engineering approach was utilized. The domain experts, high pressure oxidizer 
turbopump specialists, were queried as to how they review the time history 
plots of sensor data that are generated from the x-y time history plots 
discussed above. Ihe manual review of these plots is the normal procedure for 
both successful tests and premature shutoff. By analyzing these plots, the 
experts can determine the operating characteristics of the engine components. 
The function of the knowledge engineer is to compile logic flow graphs of this 
review and analysis methodology. A complete listing of all of the logic flow 
graphs developed in this program is given in Appendix C. 

figure 1 of Appendix C is one such logic graph. In this instance there was a 
successful test, and the normal post-test data review procedures were 
followed. During the review process it was found that the turbine discharge 
temperature was higher than expected. The temperature had not reached the red 
line limit but had exceeded the expected operating levels. Ihe expected 
levels are determined from data stored in the test/flight data base, which is 
considered an experiential or shallow knowledge base. This data base has 
within it all past histories of relevant engine operating conditions and 
serves the purposes of table lookup. 


Ihe HMS System also provides for a deep knowledge base, lhis knowledge base 
is comprised of analytical models of the SSMt engine and the oxidizer 
turbopump. When there is insufficient or missing data within the shallow 
knowledge base, calls to this deep knowledge are made and the necessary 
information provided. For the case presented in Figure 1, it was necessary to 
access a FORTRAN coded procedure that utilizes first principals of 
thermodynamics and fluid flow to compute values for which there are no direct 
sensor measurements and to generate flow characterization coefficients. 


R I/RD89-1 I] 


Page 4 


NAS3-25279 
HP01P Pinal Report 


As can be seen, both the deep and shallow knowledge bases are resident within 
the system and assume an active role only when needed in the logic flow. At 
each node in the logic tree the expert trys to determine the cause of the 
anomaly. If successful the search ends, else there is a logical progression 
to the next node. By logically exhausting all possible alternati ves , it was 
concluded, for this case, that there was a turbine tip seal problem. 

Algorithm development followed the definition of the logic flow graphs. Each 
box or node of the logic graphs represents the heuristics which the domain 
expert uses in performing pump analysis. Since they follow in a sequential 
manner they readily lend themselves to a procedural approach, lhe process of 
developing the algorithms therefore involved assigning numerical values to 
each decision point and defining a methodology for the logical progression 
through the graph. 

Several methods were employed to assign the numerical values. Data values 
that are constants were entered as assigned variables. Where sensor data was 
missing, analytical models were used to provide the necessary information, 
lhis method was discussed above relative to flow coefficients. Ihe final 
method made use of database values or table look up. The limits of this 
program did not permit establishing a relational database for values, such as 
assembly information. However, the algorithms are so structured that future 
activity will easily permit the incorporation of database values through 
program calls. This is discussed further in the HMS development plan. 

In addition to providing for the analysis of sensor data, a methodology was 
also established for detecting and handling faulty sensor values. Appendix D 
is a replication of several algorithmic approaches for validating sensor input 
that appeared in Monthly Status Report A, 22 March 1988. lhis report 
discusses both hard and soft sensor failures as well as an advanced technique 
based upon diagnostic expectations. Ihe HMS Development Plan discusses how 
these techniques can be implemented in the next generation turbopump 
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diagnostic system. There are several methodologies for information validation 
that are currently employed by the domain experts when reviewing the sensor 
data. Where applicable and practical these have been incorporated into this 
program. Coherence techniques utilize redundant channels of the same 

measurement: to compare for similarity. Measured values are also compared to 
limit values, or end points, that a properly functioning sensor is capable of 
providing. The same measured values can be recorded prior to engine start and 
after engine shutdown for computation of differentials and drift. Finally, 

the values can be compared to what t fie fundamental laws of physics would 

dictate are possible, as in the case of mass and energy continuity. Ihis 

methodology is a portion of the function of the deep knowledge base within the 
knowledge system. In developing the algorithms those procedures just 

discussed that could be coded were incorporated into the diagnostics of this 
program. 

Ihe technical activities in lask were the coding of the algorithms and the 
creation of an HMS Development Plan. Rocketdyne made the decision to develop 
a functional, prototype system to demonstrate the diagnostic capabilities of 
the system for its interim program review at NASA Lewis Research Center. Ihe 
system was developed using the "C" programming language. This language was 
chosen because the source and executable code are deliverable and usable by 
the customer without the need for extra software purchases, and the data 

driven, forward chaining nature of the diagnostic system was readily 
implemented in a common procedural language such as "C". As program 
development continued, the demonstration system evolved into a completely 
functional, diagnostic system. This system is considered to be a deliverable 
item within this program and will be demonstrated at the final program 
review. Five data files will also be delivered, one for each of the failure 
modes that have been observed during turbopump operation. These data files 

will be used to demonstrate the capabilities of the system. Iwo failure 
modes, primary turbine seal arid primary oxidizer seal wear, have never been 
observed during operation of the current HP01P design and, therefore, data 
files do not exist for them. 
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In addition to the Rocketdyne computer code, program team member University of 
Alabama at Huntsville is using the information provided by Rocketdyne to 
develop, in a parallel effort, a comparable diagnostic system using the expert 
system development tool G2. This system will also be delivered as a part of 
this program. By having the diagnostic capability represented in two 
different formats, conventional language and expert system shell, trades can 
be made as to the direction for future development efforts. 

An HMS Development Plan was created and delivered as part of Monthly Report 
13, 16 January 1989. Ihis plan is included as Appendix E. It presents a 
descriptive methodology for expanding the system, particularly the logic flow 
graphs and ensuing code, created during this program. Areas for inclusion in 
subsequent programs would include: multiple failures and failure propogation; 
transient analyses; power level changes and throttling; an expansion of the 
deep knowledge base; and database identification and development. Since each 
of these areas is formidable, a phased development program was proposed, 
figure 2 of the Development Plan is a pictorial representat i on of the 
envisioned expert system. By utilizing the same development procedures as 
used in this contract the system presented in the conceptual design can be 
systematically defined, modeled, and developed. 


CONCLUSION 


A Health Monitoring System for the SSME HPO'IP was defined, modeled, and 
developed. The system captures the knowledge that the domain experts utilize 
in performing post test/flight data analysis. The knowledge was encoded as 
part of a knowledge based system that automates this analysis procedure. The 
system was demonstrated during an interim program review by processing data 
files containing SSME HP01P hot-fire test results. In addition to developing 
the diagnostic system, a Development Plan was created that, identifies areas 
for future effort and prescribes a sequential procedure for accomplishing 
these objectives. Ihe completion of this program is a first step in the 

development of a universal pump health monitoring system. 
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APPENDIX A: TABLE 1 


Degradation modes and corresponding PID numbers 


Primary Turbine Seal Wear 

Primary Measurement: 990 

Secondary Measurements: 2, 233, 234, 1190, 63 


Secondary Turbine Seal Wear 

Primary Measurement: 91, 92 (red lines) 

Secondary Measurements: 2, 233, 234, 990, 937, 1100, 63 


Turbine Interstage Seal / Tip Seals Wear and Erosion 
Primary Measurement: 233, 234 (red lines) 

Secondary Measurements: 2, 8, 2176, 63, 231, 232, 334 

1949, 1994, 1996, 1998, 1952, 1961, 1962 


Intermediate (purge) Seal Wear 

Primary Measurement: 211, 212 (red lines) 

Secondary Measurements: 233, 234, 937, 1100, 1188, 1190 


Primary Oxidizer Seal Wear 

Primary Measurements: 951, 952, 953 
Secondary Measurements: 2, 937, 1100, 1187 


Pump Impeller / Turning Vane Cavitation Erosion 
Primary Measurement: 2 

Secondary Measurements: 8, 2176, 63, 90, 190, 334 


Ball Bearing Wear 

Primary Measurements: 1949, 1994, 1996, 1998, 1952, 1961, 1962 
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PID Number Description 


2 


HPOTP Shaft Speed (accelerometer) 

8 


Mixture Ratio (Indirect, Flight calc) 

2176 


Mixture Ratio (Facility, PID number varies) 

63 


MCC Pc 

287 


Pc Reference (commanded) 

3001 


Power Level (Based on PID 63, PID Number Varies) 

90, 190, 334 

HPOP Discharge Pressure 

91, 92 

Secondary Turbine Seal Drain Pressure 

211, 

212 

Intermediate Seal Purge Pressure 

231, 

232 

HPFT Discharge Temperature 

233, 

234 

HPOT Discharge Temperature 

937 


Intermediate Seal He Purge Pressure, Upstream 
of PCA Orifice 

1100 


Intermediate Seal He Purge Drain Temp 

951, 

952, 953 

Primary Oxidizer Seal Drain Pressure 

990 


Primary Turbine Seal Drain Pressure 

1187 


Primary Oxidizer Seal Drain Temperature 

1188 


HPOT Secondary Seal Drain Temperature 

1190 


HPOT Primary Seal Drain Temperature 

1949, 

1998 

1994, 1996, 

PBP Radial Accelerometers 

1952, 

1961, 1962 

Turbine Radial Accelerometers 
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APPENDIX B 

DATA PROCESSING AND SIGNAL CONDITIONING 



During operation of the SSME, 85 engine parameters are 
monitored by the SSME controller. At 40ms intervals (100ms 
intervals during engine preparation and post-fire phases) , 
the controller sends a block of 128 digital values known as 
a vehicle data table (VDT) consisting of the 85 digitized 
measurements, 12 calculated parameters, and 31 redundant 
parameters and miscellaneous control words. The numeric 
values of the VDT are passed from the engine through the 
Vehicle Engine Electrical Interface (VEEI) . During flight, 
the shuttles General Purpose Computer (GPC) acts as a data 
acquisition system. The 128 word VDTs are stored in the 
onboard continuous-loop recorders and are telemetered to 
NASA's own computer system which, in turn, may be accessed 
by Rocketdyne through electronic tie-ins. 

During hot- fire engine testing, the Command and Data 
Simulator (CAD) takes the place of the GPC as the data 
acquisition system. The VDTs are passed through to a 9 
track 1600 bpi magnetic tape recorder. In addition to the 
CADs system, the Facility Recording System also operates 
during hot-fire testing. This system samples 300 
parameters at 20ms intervals. The 300 parameters consist 
of test facility measurements and osme engine parameters 
along with a number of redundant measurements. The Facility 
Recording System performs limited redline checking (in 
addition to that done by the SSME controller) for possible 
engine shutdown. The data is stored on a second 9-track 
magnetic tape in raw measurement form, i.e. milivolt values 
and raw counts. The digital tapes are read by a Perkin- 
Elmer computer system, converted into 32 bit packed binary 
floating point representation, and stored on a 9-track tape. 
Computer programs are then used to selectively extract data 
and perform a variety of manipulations. Results can be make 
available in several forms including time-history plots of 
sensor values and magnetic tapes of sensor data vs. time. 
Figure 1 is a diagram of this signal processing procedure. 

Separate from the digital recording system is an analog 
recording system in which the data from certain 
measurements, including that from accelerometers and strain 
gauges, are stored in contimuously-varying analog voltage 
form on Frequency Modulated (FM) tapes. Rocketdyne 's 
processing of this data consists of converting the analog 
signals into 10240 digital signals per second. A FFT data 
reduction technique is applied to the data resulting in 
Fourier coefficients characterizing the signals in the 
frequency domain. The sensor measurements, in this form, 
are stored on 1 GByte, 12 inch optical disks. Further 
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processing typically includes the generation of Power 
Spectral Density (PSD) plots, giving the square of 
acceleration divided by frequency versus frequency. These 
plots show the magnitude of vibration sensed at varying 
frequencies, including the resonant peaks corresponding to 
the vibration modes present within the signal, and can also 
aid in the identification and categorization of 
unrecognized peaks. Other output formats include plots of 
frequency spectrum vibration trends over the duration of a 
test, and RMS acceleration values over time. This data can 
be made available as x-y point values, the format for 
plotting, stored on 9 track 1600 bpi magnetic tape. See 
figure 2 for a diagram of the analog signal conditioning 
and processing. 
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To Host 
Computer 


Figure 1: Data Processing and Signal Conditioning 
of Digitally Recorded SSME Sensor Data 






SSME TEST SITE 




Figure 2: Processing and Signal Conditioning of SSME 
High-Frequency Analog Test Data 
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SSME 
Test Data 



Test Abnormally 


Test Failure 

no 

Terminated? 

yes 

Analysis 


(l i 

r 

Breadth First Scan 

* 

I Normal 

Post-Test 


Through All Time 


Anamolous 

Data Review 


History Plots of 


Condition(s)? 


I 

End 

Search 


yes 


Turbine Discharge 
Exaust Temp. 
Exceeds Normal 
| Bounds for 100% RPL| 
P233.234 


I 


[Diagnose Problem 


I 


Check Engine 
Influences 


Temp. Consistent With 
Propellant Mixure Ratio?J 
P 2176 


yes 


Are Internal Engine 
Losses Causing Pump 
To Run at a Higher 
Power Level? 
HPOTP DISCH PR, 
P334 


yes 


■© 


Test/Flight Database 
! Establishing Expected 
Operating Levels 


Is Fuel Turbine 
Discharge Temp 
Running Low? 
P231.232 


yes 


•© 


Suspect Problem In 
Pump Section of 
HPQ TP 
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Primary Turbine Seal 
Drain Pressure Outside 
Normal Range P990 


Check 

Engine Influences 


Yes 


Other 

Failure 

Modes 


Yes 


Is Pressure Consistent 
With Shaft Speed? P2 

1 F to 

Pressure Consistent With 
Turbine D/S Temps.? 
P233, 234 


|n o 


Yes 

Any Evidence Of 

Erratic 

to 


Seal Pressure? 

P99Q 



Perform 
Post-Test 
Leak Check 


Compare 
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bly Leak 
Checks 


High 


Turbopump 
Assembled 
With Large 
Shaft/Seal 
Clearance 


High 


L 


Dram Pressure 
High or Low? 


Low 


Compare Seal 
Clearance At 
Assembly 
With Normal 
Values 


No 

Change 


Low 


to 

Apprecible 
Wear 


Higher 


Confirm 
With Post- 
Test Leak 
Checks 


Probable 
Seal Wear 


Potential 

Blockage 

At 

Secondary 
Seal or 
Drain 


1 
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Blockage 
at Primary 
Seal Drain 
Downstream 
of Pressure 
Sensor 


Compare Seal 
Clearance At 
Assembly 
With Normal 
Values 


Low 




High 


Potential 

Blockage 

Upstream 

Of 

Primary 

Turbine 

Seal 


Turbopump 
Assembled 
With Tight 
Shaft/Seal 
Clearance 

X 

Confirm 
With Post- 
Test Leak 
Checks 
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Primary Turbine Seal 
Drain Pressure Outside 
Normal Range P990 


I 




E 
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Yes 

With Shaft Speed 9 P2 


^ No 

1 
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Erratic 

No 


Seal Pressure? 

P990 



Perform 
Post-Test 
Leak Check 

High 

Drain Pressure 

1 

r 

High or Low? 


Compare 
To Assem- 
bly Leak 
Checks 


Turbopump 
Assembled 
With Large 
Shaft/Seal 
Clearance 


High 


Low 



Higher 



Com pa 
Cleara 
Asse 
With r 
Val 

re Seal 
nee At 
mbly 
formal 
ues 


Low 


Compare Seal 
Clearance At 
Assembly 
With Normal 
Values 


Low 
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Seal or 
Drain 


l 
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Seal Drain 
Downstream 
of Pressure 
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Blockage 

Upstream 

Of 

Primary 

Turbine 

Seal 



Turbopump 
Assembled 
With Tight 
Shaft/Seal 
Clearance 

X 

Confirm 
With Post- 
Test Leak 
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mode 2 


Secondary Turbine Seal 
Cavity Pressure Outside 
Expected Range. P91, 92 


Check 

Engine Influences 


Yes 


Other 

Failure 

Modes 


Yes j 


Influenced By | 
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Seal Flow 
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P937, 1100 


Yes 


No 


Yes 
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Post-Test 
Secondary 
Seal Leak 
Check 


Compare 
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bly Leak 
Checks 
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Yes 
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Shaft Speed ? P2 

No 

I 
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Temps'? P233 ( 234 


I 


No 
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Power Level ? P63 


I 


No 
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Cavity Press.? P91.92 


I 


No 


Secondary Seal Press. 
Consistent With Primary 
Seal Press. ? P990 


1 

_ _< 

No 

f 
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High 

v 
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Clearance At 
Assembly 
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Compare Seal 
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r 
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Turbo pump 
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Clearance 


I 
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Blockage 
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Seal 


Confirm With 
Post-Test 
Leak Check 


Confirm With 
Post-Test 
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mod ©4 
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L 
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Appendix D 


I . Introduction 

Any diagnostic expert system that relies on data obtained from 
physical sensors must take measures to ensure that the sensors 
themselves are functioning correctly and providing valid data. A 
number of techniques are available for the detection of both 
large (out-of-range or large bias) and soft (small bias or drift) 
errors in the HPOTP sensor instrumentation. 


II. Hard Failures 

Hard failures are naturally the easiest to detect. The simplest 
method assumes that each sensor has a plausible range of 
operation bounded by a minimum output value (r m ^ n ) and a maximum 
output value (r max ) . If the sensor output becomes lower than 
r min then it has probably failed by going "open." If the sensor 
output becomes greater than r_ ax then the sensor has probably 
"shorted." In either case the failure is easily detected by 
examination of the sensor output, either visually or by computer 
analysis. 

Both pretest and postest calibration checks are currently done to 
detect hard failures. This test applies a known quantity of a 
sensed variable (e.g., pressure, temperature, torque) to a sensor 
of the appropriate type. The output voltage of the sensor is 
then compared to a nominal curve of sensor output voltage versus 
the sensed variable. If the tested output voltage is off the 
curve by more than a given maximum tolerance then the sensor is 
malfunctioning. If the data from these tests is recorded and 
available then a computer program can easily detect such 
failures. 

For redundant sensor information, consisting of output from three 
or more identical sensors (e.g., boost pump discharge pressure is 
measured by four identical sensor channels) , voting can be used 
to detect hard failures and in some cases soft failures. The 
standard voting procedure detects a marked deviation in one (or a 
minority) of the three (or more) signals by assuming that the 
output from the majority of sensors is correct. Another method 
of combining redundant sensors is "auctioneering" which simply 
ignores the lowest or highest sensor output 1 . Again these 
methods are easily implementable in a computer algorithm. For 
example the following algorithm will detect and throw out a 
minority of redundant sensor readings that violate the range 
bounded by r min and r max : 


RI/RD89-1 71 


Page 22 


Given: 


r min' r max : as explained above, 
n : total number of redundant sensor channels. 

1. For each sensor channel (i = 1 to n), 

if >** r m ^ n AND <*= r max then mark as GOOD. 

2. Calculate Tg ■= Total number of GOOD S^'s. 

3. If T_ >« n DIV 2+1 then proceed to step 4 (DIV 
returns an integer quotient) , otherwise the sensor value is 
undetermined . 

4. Combine the GOOD S^'s into a single composite value S c 
(e.g., take some measure of central tendency such as mean 
or median) . 


III. Soft Failures 

More subtle sensor malfunctions are also algorithmically 
detectable. These are characterized by small bias errors; or 
drift errors that increase relatively slowly with time. The 
following examples will illustrate this type of failure. 

One of the existing HPOTP sensor types measures boost pump 
discharge pressure. Nominal output for this type of sensor 
during engine firing is depicted in figure 1. Figure 2 depicts 
output that drifts away from a steady norm, becoming more marked 
with time. Past experience indicates that this kind of output is 
caused by sensor malfunction rather than by actual behavior of 
the turbopump. This malfunction can be detected using the 
following simple algorithm: 

Given: 

APjuax • maximum allowable drift in average pressure. 

t lt t 2 , t 3 , t 4 : time points such that t, to t 2 

establishes initial average pressure and t 3 to t 4 
establishes final average pressure. 

1. Calculate initial average pressure 


p init 
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PRESSURE 


PRESSURE 


0 


I 



I 



TIME 


t 3 t 4 521 


Figure 1 - Nominal Pressure Sensor Output 



Figure 2 - Malfunctioned Pressure Sensor Output 
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2 . 


Calculate final average pressure 


3. 



p final 

Calculate change in average pressure 


t 4 - t 3 + 1 


A P - **> s ( p final - p init> 

4. If AP > AP Inax then the sensor has malfunctioned, 
otherwise assume the sensor is correct. 


The input to this algorithm is assumed to be digital in form with 
a high enough sampling rate to be representative of the original 
analog pressure reading. 

Another example of soft failure detection is illustrated by the 
torquemeter, a proposed future sensor for the HPOTP. Again we 
compare nominal sensor output to anomalous output that is known 
to indicate sensor malfunction. Figure 3 depicts nominal AC 
voltage output from a torquemeter and figure 4 depicts output 
from a malfunctioning torquemeter. The waveform in figure 2: is 
missing the "B" peaks present in the nominal waveform. This 
difference can be detected by the following algorithm: 

1. Determine initial baseline voltage V^. 

2. Determine t A = time of occurrence of the first peak 
greater than (an "A" peak) . 

3. Determine t B = time of occurrence of the next peak 
after t A greater than V b (the "B" peak in a nominal wave) . 

4 . Calculate T AB *= t B - t A . 

5. Determine t 3 = time of occurrence of a peak 
less than V fa (a "negative" peak). 

6. Determine t^ * time of occurrence of the next peak 
after t 3 which is less than Vjj. 

7. Calculate period P - t 2 - t 3 . 

8. If T ab ^* P then the sensor has malfunctioned (i.e., the 
"B" peaks are missing so the time between successive 
positive peaks is close to the period length) , otherwise 
assume the sensor is correct. 


Again the input is assumed to be digital with a high enough 
sampling rate to capture the significant peaks in the data. 
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VOLTAGE 


X> 



TIME 

Figure 3 - Nominal Torquemeter Output 


VOLTAGE 



TIME 

Figure 4 - Malfunctioned Torquemeter Output 
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These methods demonstrate the ability to detect common types of 
soft sensor failures through the use of simple algorithms. Many 
soft failures in other types of sensors can be detected using 
this kind of approach. 


IV. An Advanced Failure Detection Method 

An alternative approach to failed sensor detection relies on the 
idea of "diagnostic expectations" 2 . An expert in the field of 
turbopump diagnosis has certain expectations about the 
characteristics of a final answer. These expectations provide a 
basis for judging the validity of the derived answer. For 
example consider the following values for some of the HPOTP 
sensors: 


Variable Status 

Boost Pump Discharge Pressure Low 

Boost Pump Discharge Temperature High 

Turbine Discharge Temperature High 

Shaft Speed Normal 


Here the boost pump discharge pressure and temperature are 
immediately suspect since temperature and pressure are normally 
proportional to each other and not inversely related. We surmise 
that one of the two sensors may be malfunctioning, but which one? 
On further examination we find that the turbine discharge 
temperature is abnormally high and the shaft speed (as measured 
by torquemeter) is normal. An expert might conclude that the 
boost pump pressure reading is probably incorrect since the high 
turbine temperature corroborates the high boost pump temperature 
and the shaft speed does not contradict this conclusion since it 
is not abnormally low. 

A particular turbopump condition can be inferred from a certain 
pattern of sensor readings. In this case the sensor pattern does 
not fit any plausible turbopump condition, and corroboration and 
correlation between sensor values reveals that the boost pump 
pressure value is probably incorrect, meaning that the pressure 
sensor has failed. 

The partial decision tree in figure 5 captures this line of 
reasoning. This kind of decision tree can be easily implemented 
in Prolog or in one of the expert system shell languages. 

Uncertain reasoning can be incorporated by associating a certainty 
factor or weighting factor with each branch of the tree. Of 
course this is a simple and incomplete example, but it 
illustrates the method of expectation-based 6ensor validation. 
No doubt there are many such correlated sensor patterns among the 
current and proposed HPOTP instrumentation and these will be 
revealed by further interviews with sensor experts. This method 
can be used by itself to detect sensor failure, or to verify 
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Boost Pump 
Pressure 



Figure 5 - Decision Tree for Sensor Validation 
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sensor failures that were initially detected using other methods. 
V. Conclusion 

Many techniques exist for the detection of both hard and soft 
failures in HPOTP sensors. These methods can be expressed as 
algorithms and thus are implementable as computer programs; or 
expert systems. Sensor validation for the proposed HPOTP health 
monitoring system will be accomplished using the following 
methods : 

1. Detection of hard failures using range checks and 
voting where appropriate. 

2. Detection of soft failures where possible using a 
straightforward algorithmic approach as described in 
section III. 

3. Verification of soft failures detected above by 
corroboration and correlation with other related sensor 
values . 

4. A general expectation-based examination of sensor 
values to identify anomalous readings and determine which 
of these is caused by sensor failure. 


Once sensor readings have been validated using these methods, the 
task of actual HPOTP fault diagnosis can proceed with confidence. 
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APPENDIX E 


HMS DEVELOPMENT PLAN 


Development of the HMS will follow the methodologies and 
procedures that were used in satisfying Contract NAS3-25279: 
Reusable Rocket Engine Turbopump Health Monitoring System. 

The discussion will follow the analysis that was used for 
the HPOTP, as shown is Figure 1; however, the technologies 
embodied within approaches presented are directly 
transferable to any engine component or to the entire 
engine as a functional unit. 

Initial activity in the development of any Health Monitoring 
System is the identification of the failure mode/sensor 
system couplets. For the existing SSME HPOTP, these 
couplets have been defined in great detail and are 
explicitly given in the above mentioned report. This list 
will be continuously reviewed to account for any changes in 
pump or engine design which may have an influence upon the 
number and type of failure modes that are to be analyzed as 
well as the number and type of sensors that provide 
information relative to the engine component. Changes in 
this category are expected to occur very infrequently and 
can only be included within the diagnostic system after 
extensive testing and verification by engineering test and 
quality assurance. However, recognizing that some changes 
will occur, the computer system, software diagnostics and 
prognostics, will be designed and developed such that any 
changes and/or modifications can be easily incorporated. 

In the same manner, it is expected that the data processing 
and signal conditioning will change very little. 

Additional processing may be required if new or additional 
sensors are added to the system. The method by which this 

data is collected and processed is not part of the HMS; 
rather, the end product, the magnetic tapes, is of concern 
in this diagnostic system. The HMS development team can 
ask that the tape data be in a certain format for easier 
computer uptake; however, actual data collection and 
reduction is outside of the scope of this program. 

The information provided by existing sensors may prove to be 
insufficient in helping identify the occurrence of certain 
failure modes. For such cases, alterations to the sensor 
complement or to sensor data processing may be specified. These 
alterations may include relocation of sensors nearer to the 
physical location of the failure mode, thereby decreasing the 
localized component configuration effects. An example of this 
type of effect is pressure drop due to flow through a length of 
drain manifold. In addition to sensor placement, additional 
sensors may be prescribed for specific measurements relative to 
individual failure modes. An example of this would be plume 


RI/RD89-1 71 


Page 30 


spectroscopy for wear detection. Many failure inodes involve 
either seal wear, bearing wear, or cavitation erosion. Use of 
detectable compounds and isotopes embedded in key locations of 
likely wear can give indication of single or multiple failure 
modes . 

Further investigation into the nature of information derivable 
from high frequency analog measurements (such as accelerometers 
and strain gages) may also be desirable. For example, it may be 
possible to identify a particular bearing as undergoing 
significant wear by specialized phase or amplitude analysis of 
data from various strain gages or accelerometers. 

Figure 2 is a cross-sectional line drawing of the space shuttle 
main engine high pressure oxidizer turbopump. Surrounding the 
figure are the names if several HPOTP parameter sensors used 
for the diagnostic and prognostic functions of the Reusable 
Rocket Engine HMS (mentioned above) . Note that aside from the 
accelerometers and strain gages (some of which are not present 
in the pump's flight configuration) and the secondary turbine 
seal cavity pressure, all the sensors are somewhat removed 
from the region of their associated failure mode. They are 
often located up- or down-stream of the pump at its connecting 
flanges. In fact, one section of the turbopump HMS deep 
knowledge base performs extrapolation of fluid property values 
from the physical region of the sensor to nearer the point 
of interest, such as the actual discharge region of a pump or 
turbine. 

Once the sensor/failure mode couplets have been identified 
the process of defining the diagnostic and prognostic 
analysis logic begins. These flow graphs are the 
sequential logical steps by which the domain experts, eg. 

HPOTP diagnostic experts, analyze the time history plots of 
the sensor data to determine if any anomalies are present. 

For Contract NAS 3 -2 52 79, seven flow graphs were generated, 
one for each of the primary failure modes of the turbopump. 

Each primary mode constitutes a single point, uncoupled 
failure which would occur during steady state operation. 

Changes in these graphs would need to be made only when new 
sensor information becomes available or when the character 
of the failure modes themselves change. A complete listing 
of these graphs was delivered to NASA Lewis Research Center 
during the first program review on 1 December 1988. 

A major element within the HMS Development Plan would be the 
expansion of the logic flow graphs now in existence. There 
are several prominent areas in which this can occur. These 
include: 


1. multiple failures occurring simultaneously or those 
generated as a result of another failure, failure mode 
propagation, both within the turbopump as well as those 
caused within the pump as a result of a failure within 
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another component of the engine 

2. transient condition analysis, both start and 
shutdown 

3. power level changes, throttling 

4. sensor data validation and procedures to reconstruct 
or provide data when it is missing 

5. existing analytical models, deep knowledge. 

6. database identification and development 

Each of these tasks is formidable by itself and as a group 
constitute a major development effort. As an example, the 
transient conditions are poorly understood and require 
particular attention if they are to be included within an 
HMS. Therefore, caution must be exercised when 
establishing the goal for the HMS and the time course for 
its development. For this reason, a sequential development 
program is proposed for the HMS. In the first phase, the 
existing logic flow graphs will be expanded to accommodate 
power level changes and sensor data validation and 
reconstruction. Also during this phase information will be 
gathered and assimilated relative to multi-point failures and 
failure mode propagation, data base development of 
historical engine and test data, and existing analytical 
models that can be used to provide information relative to 
the failure modes. The data base of assembly , fluid flow 
test, and statistical data will be accessed as part of the 
information used by the inference engine while the 
analytical models will form the deep knowledge base within 
the expert system. These two knowledge bases will be 
discussed in more detail later is this report. The second 
phase of the development plan will be to develop new logic 
flow graphs, where warranted, that account for multi-point 
and failure mode propagation. In addition, both the shallow 
and deep knowledge base will be incorporated into the expert 
system at this time. Also during this phase, investigation 
will begin of the transient cases. The third phase of the 
program will be to finalize all of the logic flow graphs, 
including those for the transient cases. With this, the 
logic by which engine anomalous conditions, from engine 
start to stop, will have been identified and developed. 

As shown in the HMS Conceptual design, Figure 1, following 
the development of the logic flow graphs is algorithm and 
heuristics development. The distinction between the graphs 
and the algorithms/heuristics is that numerical and data 
values as well as all subroutine computer calls are 
identified in the algorithms while inferential and rule of 
thumb methodologies constitute the heuristics. Development 
here will follow the same phase format as defined above. 
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It is intended in this development plan that any time the 
program calls for the development of a logic graph or 
algorithm or analytical model that this will also be 
incorporated into the expert system to perform relevant 
diagnostics and prognostics. 

For any given phase of the development, once 
algorithms/heuristics have been identified software 
implementation begins. Due to the anticipated size of the 
final HMS for any engine component or entire engine system, 
the host computer for the HMS must be sufficiently large, in 
terms of memory and processing capacity. For this reason, 
the computer chosen will be a work station such as a SUN 4. 
To facilitate the software development in terms of rule 
implementation, changes, modification, data entry and 
access, networking, and natural language structure it will 
also be beneficial to use an expert system development tool 
such as ART, KEE, or G2 . 

As the HMS Conceptual Design shows. Figure 3, the host 
computer/expert system has a multitude of components and 
operations. The system must be able to access large data 
bases, have an extensive inference engine, allow for calls 
to FORTRAN subroutines, have a user friendly interface, 
allow for fuzzy logic or reasoning under uncertainty, 
provide logic trees of its inference strategies, and be 
maintainable. The FORTRAN subroutine access is necessary 
since several existing Rocketdyne programs, eg. SCOTTY, 

SAFD, that can be utilized in this HMS, are coded in 
FORTRAN. The selection of the system that can best 
accomplish all of these objectives will be performed during 
the early portion of phase one. As soon as 
algorithms/heuristics/rules are developed they will be 
entered into the expert system. Proper selection of an 
expert system development tool (shell) will provide an 
environment that will promote ease in creating the expert 
diagnostic/prognostic system. In this manner, continuous 
implementation and testing will take place during all phases 
of the HMS program. 

As can be seen in Figure 3, Oxidizer Turbopump Health 
Monitoring Expert System Block Diagram, the Conceptual 
Diagram for the expert system consists of several 
components. Each of these components is a part of the 
development plan and will be developed and incorporated 
during the appropriate phases discussed above. The 
structure of the expert system itself, the inference engine 
and knowledge base, is provided by the tool and shell 
chosen. For this reason it is not necessary to talk about 
developing the inference engine or knowledge representation 
format, but rather implementing the knowledge, facts, and 
rules embodied within the logic flow graphs, algorithms, and 
heuristics into a software system. Once the particular 
language of the tool is mastered, it becomes a straight 
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forward procedure to perform the actual coding. 

The function of the data base is to provide a store of 
knowledge about assembly data and historical data relative 
to a particular engine, test, and/or component. This 
information is resident within Rocketdyne and is collected 
for every engine and component that is built. Development 
of the data base for purposes of this program will be to 
structure this data into a relational or other relevant 
format such that it can be accessed by the inference engine 
on an as needed basis. It is essential that the expert 
system tool selected be capable of addressing large data 
structures . 

There are several diagnostic tools available at Rocketdyne 
that may provide relevant information for the Health 
Monitoring System developed in this program. These include 
SAFD, SCOTTY, and ADDAMX . All of these tools are written 
in FORTRAN; therefore, in developing the HMS system, 
software procedures must be defined and developed that can 
access these routines when needed. This does not pose a 
serious development problem since most expert system 
development tools have a built in capability to address such 
programs. However, one group of analysis tools that exist 
at Rocketdyne that will most likely require task specific 
modification are the analytical models that will form the 
deep knowledge base within the expert system. 

Several of the analytical models that will be considered for 
inclusion within the deep knowledge base are the power 
balance model, the aero-thermo model, HPOTP component 
analytical models, and the life prediction models. These 
models are now in existence at Rocketdyne and are in use on 
other programs; however, they have never been coordinated 
into a unified system. Their function within the knowledge 
base is to provide a second source of information and 
verification where there may be gaps in sensor data, to 
perform diagnostic analysis independent of the expert system 
thereby allowing for internal validity checks, and to 
provide the basis from which prognostic analyses are made. 

It is not the intention of the HMS Development Program to 
alter or develop these models, in terms of their constituent 
analysis functions, but rather to modify, if needed, their 
format such that they can be incorporated into the HMS. 
Shortfalls in analysis capability relevant to the HMS will 
be identified and corrective procedures suggested such that 
the relevant groups within Rocketdyne can begin to make 
needed changes and/or modifications. 

As mentioned above, as each phase of logic 
flow/algorithm/heuristic development progresses this 
knowledge is entered into the rule and control structure of 
the HMS. Since each phase of the HMS development process 
has a specific output, in terms of diagnostic and prognostic 
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capability, it will also be possible to validate the 
software code in parallel with this development. The first 
step in the validation process will be a review of the logic 
graphs, desk audit, to check for inter- and intra-mode 
diagnostic consistency. Following this 'hand check', 
validation will consist of execution of the software with 
all relevant hot fire engine component data. Test and 
flight data exists for every engine and component developed 
under the SSME program. By methodically running this data 
through the program and verifying the computer output by 
domain experts a major step in the validation will have been 
accomplished. Inconsistencies will be corrected and the 
development will then proceed. Software validation will be 
an ongoing process through all phases of the program. With 
final Rocketdyne certification of the software, the HMS 
system will be demonstrated at NASA Lewis. Upon 
acceptance, the system will then be delivered to the 
Research Center. It is the intention of this development 
program that Rocketdyne help NASA maintain the system by 
incorporating any modifications or changes resulting from 
new capabilities and/or engine changes. 


SUMMARY 

An HMS conceptual design and development plan has been 
presented which will provide a complete HPOTP, or total 
engine, between flight, diagnostic and prognostic expert 
system. Development will follow a sequential approach 
whereby at each successive level of development greater 
analysis capability and sophistication is added to the 
system. The final expert system will have both a shallow 
and deep knowledge base, access existing diagnostic programs 
as needed, be capable of maintaining and addressing large 
databases of information, provide a user friendly interface, 
and be easily maintainable. Upon completion of the 
development process, the entire system will be delivered to 
the NASA Lewis Research Center. 
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Turbopump Health Monitoring System Scoping 
and Solution Data Requirements 
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Failure Modes & 
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Degredation Mechanisms 
Selected Failure Modes Relating to HPOTP 
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Intermediate Seal He Purge P at PCA 
Intermediate Seal He Purge P upstream of PCA 



FIGURE 2 











FIGURE 3 

Oxidizer Turbopump Health Monitoring 
Expert System Block Diagram 
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